System and method for painting 3D models with 2D painting tools

ABSTRACT

An 2D layer containing texture coordinate information is inserted into a layered image snapshot of a 3D scene perspective. The layered image snapshot can be painted using 2D painting techniques and then imported back into the 3D scene by using the texture coordinate information to map modified pixels back into the 3D scene.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to 3D graphics renderingsoftware and, more specifically, to using 2D paint software tools topaint 3D scenes.

2. Description of the Related Art

Current 3D rendering applications provide the ability to paint a 3Dscene using 2D painting tools. Specifically, 3D rendering applicationsprovide the capability to generate a 2D image of a 3D scene from anyparticular perspective. The 2D image can then be imported into a 2Dpainting application to perform the painting. Once the painting iscompleted, the modified 2D image needs to be imported back into the 3Dscene. This is achieved by projecting the modified 2D image back intothe 3D scene from the same perspective at which the 2D image was firstcaptured.

While such techniques enable users to utilize sophisticated 2D paintingapplications, they also constrain the state of the 3D renderingapplication during the use of the 2D painting application. For example,once the 2D image of the 3D perspective has been generated, the 3Dperspective cannot be modified in any way. Any movement of objects inthe 3D perspective or change of camera position, field-of-view, or evenresizing of the software window, would result in improper alignment ofthe modified 2D image upon re-projection. Furthermore, since the 3Drendering application must stay locked in the captured perspective, onlyone generated 2D image can be created and painted using 2D paintingapplications at any particular moment of time. Additionally, dependingupon the implementation of the 3D perspective capturing technology, theresolution of the resulting 2D image may be limited to the resolution ofthe monitor. Finally, any new painting must be applied twice, once inthe 2D painting application, and again when it is projected back intothe 3D model.

SUMMARY OF THE INVENTION

One or more embodiments of the present invention provide the capabilityto paint a 3D scene using 2D painting applications while simultaneouslyallowing the 3D scene to be modified, reoriented, or repositioned andmultiple snapshots to be taken during the 2D painting process. This isachieved by preserving texture coordinate information in a generated 2Dimage of a perspective of a 3D scene by embedding a special hidden andlocked layer containing such coordinate information, such as UVcoordinates, into the 2D image. When the 2D image needs to be importedback into the 3D scene, this special layer is referenced to update therelevant pixel values of the texture maps in the 3D scene.

A method for painting a 3D scene according to one embodiment of thepresent invention includes the steps of capturing a 2D snapshot of aperspective of the 3D scene and inserting a coordinate layer into the 2Dsnapshot wherein the coordinate layer comprises texture coordinateinformation. Such texture coordinate information, for example, may be UVcoordinate values (as further described in the Detailed Description)that are embedded in the R and G (i.e., red and green) components of thepixel values of the coordinate layer. Inserting the UV coordinate valuesin an additional layer of the 2D snapshot stores such information in aformat (i.e., a 2D image “layer”) that is compatible with existing 2Dpainting applications. The method further includes the steps ofimporting a modification of the 2D snapshot, wherein the modification isgenerated by 2D painting techniques (e.g., made available through suchexisting 2D painting applications), and modifying at least one texturemap of the 3D scene in accordance with said modification.

BRIEF DESCRIPTION OF THE DRAWINGS

The file of this patent contains at least one drawing executed in color.Copies of this patent with color drawing(s) will be provided by thePatent and Trademark Office upon request and payment of the necessaryfee.

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1A depicts a structure of a 2D layered image file.

FIG. 1B illustrates a representation of a 2D layered image with 2 layerscontaining surface appearance attributes.

FIG. 2 illustrates a representation of the components of a 3D scene.

FIG. 3 depicts a flow chart of the steps taken to paint a portion of a3D scene in accordance with one embodiment of the present invention.

FIG. 4A depicts one embodiment of layers in a 2D layered image filegenerated from the steps of FIG. 3.

FIG. 4B is a pictorial representation of one embodiment of layers of a2D layered image generated from the steps of FIG. 3.

FIG. 5 is a block diagram of software components that may be executed bya processor of a computer system in accordance with an embodiment of theinvention

DETAILED DESCRIPTION

FIG. 1A depicts a structure of a 2D layered image file 100. Image file100 comprises a plurality of layers 105 _(A) . . . 105 _(N) which, whensuperimposed upon each other, comprise a final image. Each layer is abitmap or array of pixels with each pixel corresponding to a renderedcoordinate in the final image. Each pixel is an RGBA (red, green, blue,alpha) data value which, for example, may be represented by 32 bits,with each of the red, green, blue and alpha components comprising 8bits, resulting in a possible 16.7 million color possibilities for thepixel, with the alpha value indicating how transparent that pixel shouldbe. For example, FIG. 1B depicts an image 110 with 2 layers. One layer115 contains color information in each pixel value while the other layer120 contains shading and shadowing information in each pixel value. Itshould be recognized that the foregoing is merely illustrative of thebasic structure of layered image files and that known layered image fileformats such as TIFF or PSD and other proprietary formats may includeadditional headers, tags and other meta-data information.

In contrast, a 3D scene file comprises a collection of 3D polygon modelswith each such model having a polygon mesh and a set of texture maps. Atexture map is a single-layered image file that contains informationabout a surface appearance attribute of the model, such as its color,reflectivity, glossiness, specularity, or even invisible properties suchas its hardness. A polygon mesh is a collection of vertices, edges andfaces that make up the polygons which define the shape of a 3D model.Each vertex of a polygon maintains five attributes that describelocation information, which include its XYZ position in Cartesiancoordinate space and a “UV coordinate” on the surface of the model. Whenmoving, editing, or rotating a model, the XYZ locations of the verticeswill be modified to reflect their new positions, but the UV coordinatesremain constant since they are defined relative to the surface of the 3Dmodel (as further discussed below).

The 3D scene file also further comprises one or more light and cameracomponents. A camera component of a 3D scene file contains a descriptionof a location, direction and field-of-view. It is the camera componentof a 3D scene file that is accessed and modified to change theperspective of a 3D scene in a display. Similarly, a light component ofa 3D scene file contains a description of how a particular light sourceilluminates the objects in the scene (e.g., color of light, direction oflight, brightness of light, light decay characteristics, etc.). Whenrendering a 3D scene file, 3D rendering software utilizes informationfrom the camera components (which describe the location and directionfrom which the models are viewed), the texture maps (which describe howthe surface of the model should appear) and the light components (whichdetermine how the surface should be shaded and shadowed) to create animage. The models of the scene can then be shown from any perspective bymoving the camera in the scene.

FIG. 2 illustrates a representation of the components of a 3D scenecomprising a single polygon model, namely a bust. Perspective 200depicts the 3D scene from a particular camera point of view, with allthe textures (e.g., color, etc.) of the surface of the bust model inplace, shaded by the lights in the scene. The camera component of the 3Dscene file is represented by camera 205 and the light components of the3D scene file are represented by lights 210. The polygon mesh of thebust model is represented by 215. UV texture map 220 is a representationof the color of the surface of the bust polygon model that has been“unwrapped” into a 2D representation. In the embodiment of FIG. 2, “U”and “V” are the names of the axes along the 2D plane of UV texture map220. Each UV coordinate references a pixel value (e.g., divided intored, green and blue components) in UV texture map 220 that defines thecolor for the corresponding point on the surface of the 3D model andeach point on the surface of the 3D model has a corresponding UVcoordinate in UV texture map 220. When rendering the models in a 3Dscene, 3D rendering software determines for each pixel in the rendering,what surface point on which particular model in the scene is beingdisplayed (in FIG. 2, for example, there is only one model, namely thebust). Specifically, it determines the UV coordinate associated with theparticular point in the polygon model and references the UV texture mapassociated with the model to obtain the color of the pixel. It should berecognized that such UV texture maps may include further informationrelating to other surface appearance attributes, such as glossiness orreflectivity, or separate texture maps may be used to store suchadditional surface appearance attributes. By unwrapping a polygon meshof a 3D model such as 215 into a 2D texture map such as 220, artists aremore easily able to paint the surface of the 3D model in 2D, therebyassigning pixel values to UV coordinates in the UV texture map thatcorrespond to points on the surface of the 3D model. It should berecognized that a scene file may be comprised of separate files for itscamera and light components, polygon meshes and texture maps inalternative embodiments.

FIG. 3 depicts a flow of steps taken to paint a portion of a 3D scene inaccordance with one embodiment of the present invention. In step 300, a3D scene having at least one 3D polygon model comprising a polygon meshand at least one UV texture map (e.g., color texture map), is displayedby 3D rendering software. In step 305, the 3D polygon scene is rotated(e.g., by a user) to a desired perspective and a 2D “snapshot” of theperspective is requested. In step 310, the 3D rendering softwareidentifies the points on each of the 3D polygon models in the scenecorresponding to pixels that are displayed in the desired perspective,and determines the corresponding UV coordinates in the UV texture mapsof the corresponding 3D polygon models for each of these points. In step315, the 3D rendering software accesses the various UV texture maps(e.g., corresponding to a surface appearance attribute such as color,reflectivity, glossiness, etc.) for each of the 3D polygon models in thescene and generates an image layer that contains the values for each ofthe surface appearance attributes for the displayed pixels of theperspective. In step 320, each such generated image layer is given aname that corresponds to the UV texture maps for the same surfaceappearance attribute. In step 325, the 3D rendering software generatesan additional image layer comprising shading and shadowing informationfrom the perspective. In step 330, the 3D rendering software generatesan additional image layer that comprises the UV coordinates of thedisplayed pixels of the perspective (hereinafter, the “UV coordinatelayer”). In one embodiment, the U coordinate is stored as the redcomponent of the layered image layer's relevant pixel and the Vcoordinate is stored as the green component. The blue component of thepixel may be utilized to store other meta-data information. In theforegoing discussion of FIG. 3, for example, a model identifier may bestored in the blue component of the pixel to identify which model'stexture maps pertain to the UV coordinates stored in the red and greencomponents. In alternate embodiments, the blue component of the pixelmay be used to store a face identifier if different UV texture maps arebeing utilized for different faces of the model. It should berecognized, however, that the UV coordinate may be stored in anycombination of the color components of the relevant pixel. In step 335,the UV coordinate layer is tagged as hidden and locked and in step 340,the 3D rendering software consolidates the generated layers into asingle layered image file. It should be further recognized that othergenerated image layers relating to surface appearance attributes such asglossiness and reflectivity may also be set as hidden by the 3Drendering software (such that the artist may make them visible duringsubsequent painting as desired).

In step 345, the user imports the generated layered image file into a 2Dpainting application. In step 350, the user utilizes the tools of the 2Dpainting application to paint the layered image as desired (e.g., changecolors or other surface appearance attributes). In modifying the layeredimage as desired, pixel values in the various surface appearanceattribute layers (e.g., color, reflectivity, glossiness, etc.) of thefile may be modified accordingly by the 2D painting software application(but not the UV coordinate layer or the shading/shadowing layer) in step355. Upon completion of the modifications, the user imports the modifiedlayered image file back into the 3D rendering software in step 360. Instep 365, the 3D rendering software examines each pixel in each layer ofthe layered image file and, by referencing the UV coordinate (and modelidentifier, if needed) in the UV coordinate layer of the layered image,is able to appropriately update the values in each of the UV texturemaps corresponding to such layer (e.g., UV coordinates in color UVtexture map of the appropriate 3D model is updated according to colorlayer, etc.). It should be recognized that if additional coordinates areneeded in certain implementations, additional layers may be added to thelayered image file. Additionally, to the extent that a 3D polygon modeldoes not yet have an associated UV texture map, alternative embodimentsmay assign arbitrary UV coordinates to each vertex and/or point on thesurface of the polygon mesh.

FIG. 4A illustrates one embodiment of layers in a layered image filegenerated from the steps of FIG. 3. Layer 400 comprises a bitmap ofpixels representing the color of the image, with each pixel having acolor value with RGB components. Layer 405 comprises a bitmap of pixelsrepresenting the shading and shadowing of the image generated from step325 of FIG. 3, with each pixel having a color value with RGB components.Layer 410 is a UV coordinate layer generated from step 330 of FIG. 3.Layer 410 also comprises a bitmap of pixels, however, the values of eachof these pixels represent the U and V coordinate values of thecorresponding displayed pixel in the 3D polygon model. As previouslydiscussed, the U coordinate may be embedded in the R component of thepixel value while the V coordinate may be embedded into the G componentof the pixel value. While layers 400 and 405 are overlaid upon eachother to provide the final layered image, layer 410 remains hidden andlocked when displayed by 2D rendering software because it containstexture coordinate information (namely, UV coordinate information) thatis only useful when importing the layered image back into the 3Drendering scene. FIG. 4B is a pictorial representation of one embodimentof the layers of a layered image generated from the steps of FIG. 3.Final image 415 is composed of three layers: color layer 400, shadinglayer 405, and UV coordinate layer 410. Although UV coordinate layer 410can be displayed pictorially, the color pallet represented by the pixelvalues are meaningless because the UV coordinates embedded within thepixel values of the layer (e.g., U value is embedded in the redcomponent and V value is embedded in the green component) are meant tobe hidden and locked and accessible by 3D rendering software rather than2D rendering software. Indeed, as shown, final image 415 does notinclude the color overlay represented by UV coordinate layer 410.

FIG. 5 is a block diagram of software components that may be executed bya processor of a computer system in accordance with an embodiment of theinvention. A 3D software component 500 comprises a 3D renderingcomponent 505 to render a 3D model or scene (e.g., a collection ofmodels) on a display and enable the rotation of the model or scene topresent various perspectives on the display. 3D software component 500also includes a snapshotting component 510 that provides the capabilityto capture a 2D snapshot image (comprised of one or more layers witheach layer maintaining a surface appearance attribute) of a perspectiveof a 3D model or scene rendered through rendering component 505 andinsert an additional layer containing the UV coordinates into the 2Dsnapshot image in accordance with the steps of FIG. 3. Additionally, a2D mapping component 515 provides the capability to receive a layeredimage originating from snapshotting component 510 and map the surfaceappearance attribute values from the various layers of the layered imageinto corresponding UV texture maps in the 3D model or scene byreferencing the UV coordinate layer of the layered image.

A 2D software component 520 includes a 2D rendering component 525 thatcan receive a layered image file, such as one generated by snapshottingcomponent 505, and display the layered image. A user can change varioussurface appearance attributes such as the color of the displayed layeredimage by interacting with a painting component 530 of the 2D softwarecomponent. Such a painting component 530 will modify pixel values in thevarious layers of the layered image file in accordance with the actionstaken by the user (e.g., through a painting graphical user interfacepresented by painting component 530). The software components presentedin FIG. 5 may be separate or standalone software applications running ona single computer, components of a single software application, or maybe executed on a distributed system of computers communicating over adata network, such as a local area network (LAN), wide area network(WAN), or the Internet.

The various embodiments described herein may employ variouscomputer-implemented operations involving data stored in computersystems. For example, these operations may require physical manipulationof physical quantities usually, though not necessarily, these quantitiesmay take the form of electrical or magnetic signals where they, orrepresentations of them, are capable of being stored, transferred,combined, compared, or otherwise manipulated. Further, suchmanipulations are often referred to in terms, such as producing,identifying, determining, or comparing. Any operations described hereinthat form part of one or more embodiments of the invention may be usefulmachine operations. In addition, one or more embodiments of theinvention also relate to a device or an apparatus for performing theseoperations. The apparatus may be specially constructed for specificrequired purposes, or it may be a general purpose computer selectivelyactivated or configured by a computer program stored in the computer. Inparticular, various general purpose machines may be used with computerprograms written in accordance with the teachings herein, or it may bemore convenient to construct a more specialized apparatus to perform therequired operations.

The various embodiments described herein may be practiced with othercomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, and the like.

One or more embodiments of the present invention may be implemented asone or more computer programs or as one or more computer program modulesembodied in one or more computer readable media. The term computerreadable medium refers to any data storage device that can store datawhich can thereafter be input to a computer system computer readablemedia may be based on any existing or subsequently developed technologyfor embodying computer programs in a manner that enables them to be readby a computer. Examples of a computer readable medium include a harddrive, network attached storage (NAS), read-only memory, random-accessmemory (e.g., a flash memory device), a CD (Compact Discs) CD-ROM, aCD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, andother optical and non-optical data storage devices. The computerreadable medium can also be distributed over a network coupled computersystem so that the computer readable code is stored and executed in adistributed fashion.

Although one or more embodiments of the present invention have beendescribed in some detail for clarity of understanding, it will beapparent that certain changes and modifications may be made within thescope of the claims. For example, the foregoing discussions describeembodiments in which image layers are comprised of pixel values withthree color components, RGB. In such embodiments, each pixel value of aUV coordinate layer in the layered image provides storage capacity forthe U and V values (i.e., in the R component and G component, forexample) in addition to one meta-data property value, such as a modelidentifier (i.e., in the B component, for example). However, alternativeembodiments may use and/or generate layered image formats that containan additional alpha channel, resulting in an extra storage location perpixel. Because such layered image formats utilize different pixelrepresentations, they may be able to accommodate more meta-data values.Similarly, while the foregoing discussions describe embodiments in whichthe blue color channel in the UV coordinate layer may hold a modelidentifier or face identifier, alternate embodiments may store any othermeta-data information as may be needed by the 3D modeling software, suchas a “material” or “shader” identifier as further discussed below.Furthermore, such meta-data also need not correspond to an attribute ofthe pixel at the UV location. For example, before creating the snapshotof the 3D model, the user may orientate the 3D model or scene to adesired position. Parameters describing the position of the model, suchas zoom level, camera angle, etc., may be stored in the available colorvalues, such that the model can be repositioned by the 3D modelingsoftware as part of the mapping process. Similarly, although theforegoing description described embodiments having a single UVcoordinate layer to maintain meta-data such as UV coordinates, anyarbitrary amount or type of meta-data can be maintained with the 2Dsnapshot images through the addition of extra layers. These additionallayers may be locked and hidden with respect to the 2D paint softwareand are included within the 2D snapshot image file. Such additionalmeta-data can be defined as an attribute of the pixel at the UV positionas defined in the coordinate layer, or any arbitrary meta-data, whichmay or may not be associated to a particular pixel at the UV location.It should be further recognized that the techniques disclosed herein maybe utilized with 3D models that describe surfaces through means otherthen polygon meshes, such as, for example, sets of NURB patches,subdivision surfaces, implicit geometry and the like. Similarly, itshould be recognized that the techniques herein may also be utilized insituations where 3D models share texture maps (e.g., through the use of“shader” or “material” data objects, etc.) rather than having their owntexture maps. Accordingly, the described embodiments are to beconsidered as illustrative and not restrictive, and the scope of theclaims is not to be limited to details given herein, but may be modifiedwithin the scope and equivalents of the claims. In the claims, elementsand/or steps do not imply any particular order of operation, unlessexplicitly stated in the claims.

1. A method for painting a 3D scene, the method comprising: capturing a2D snapshot of a perspective of the 3D scene; inserting a coordinatelayer into the 2D snapshot wherein said coordinate layer comprisestexture coordinate information; importing a modification of said 2Dsnapshot, wherein said modification is generated by 2D paintingtechniques; and modifying at least one texture map of said 3D scene inaccordance with said modification.
 2. The method of claim 1, whereinsaid 2D snapshot comprises a plurality of layers, wherein each of saidlayers comprises surface appearance attribute information for each pixelin said 2D snapshot.
 3. The method of claim 2, wherein said surfaceappearance attribute information for each of said layers is selectedfrom the group consisting of color, reflectivity, specularity andglossiness.
 4. The method of claim 2, wherein said modificationcomprises changes to values in said plurality of layers.
 5. The methodof claim 1, wherein said texture coordinate information comprises a UVcoordinate for each pixel displayed in said 2D snapshot.
 6. The methodof claim 5, wherein, for each of said UV coordinates, a U coordinate isstored in a first color component of a pixel value of said coordinatelayer and a V coordinate is stored in a second color component of saidpixel value of said coordinate layer.
 7. The method of claim 6, whereina model identifier corresponding to a polygon model in said 3D scenewhere the pixel resides is stored in a third color component of saidpixel value of said coordinate layer.
 8. The method of claim 1, whereinsaid capturing step further comprises: accessing UV coordinates ofdisplayed pixels of said 2D snapshot; and generating at least onesurface appearance attribute layer for said 2D snapshot, wherein valuesin said surface appearance attribute layer correspond to values of saidUV coordinates in a texture map of said 3D scene.
 9. The method of claim8, wherein said capturing step further comprises naming said surfaceappearance attribute layer the same name as said texture map.
 10. Themethod of claim 9, wherein said modifying step further comprises:identifying pixel value changes made in said at least one surfaceappearance attribute layer; extracting UV coordinates from saidcoordinate layer corresponding to said identified pixel value changes;identifying said at least one texture map by matching a name of at leastone surface appearance attribute layer with the name of said at leastone texture map; and inserting said pixel value changes into said atleast one texture map at said corresponding UV coordinates.
 11. Themethod of claim 1, further comprising the step of tagging saidcoordinate layer as locked and hidden.
 12. A computer system configuredto render a 3D scene, the computer system comprising a processorprogrammed to perform the steps of: capturing a 2D snapshot of aperspective of the 3D scene; inserting a coordinate layer into the 2Dsnapshot wherein said coordinate layer comprises texture coordinateinformation; importing a modification of said 2D snapshot, wherein saidmodification is generated by 2D painting techniques; and modifying atleast one texture map of said 3D scene in accordance with saidmodification.
 13. The computer system of claim 12, wherein saidmodification is performed using a separate 2D painting softwareapplication.
 14. The computer system of claim 12, wherein said capturingstep further comprises: accessing UV coordinates of displayed pixels ofsaid 2D snapshot; and generating at least one surface appearanceattribute layer for said 2D snapshot, wherein values in said surfaceappearance attribute layer correspond to values of said UV coordinatesin a texture map of said 3D image.
 15. The computer system of claim 14,wherein said capturing step further comprises naming said surfaceappearance attribute layer the same name as said texture map.
 16. Thecomputer system of claim 15, wherein said modifying step furthercomprises: identifying pixel value changes made in said at least onesurface appearance attribute layer; extracting UV coordinates from saidcoordinate layer corresponding to said identified pixel value changes;identifying said at least one texture map by matching a name of at leastone surface appearance attribute layer with the name of said at leastone texture map; and inserting said pixel value changes into said atleast one texture map at said corresponding UV coordinates.
 17. Acomputer readable storage medium having stored therein a computerprogram for rendering a 3D scene, wherein a computer system executingthe computer program carries out the steps of: capturing a 2D snapshotof a perspective of the 3D scene; inserting a coordinate layer into the2D snapshot wherein said coordinate layer comprises texture coordinateinformation; importing a modification of said 2D snapshot, wherein saidmodification is generated by 2D painting techniques; and modifying atleast one texture map of said 3D scene in accordance with saidmodification.
 18. The computer readable storage medium of claim 17,wherein said capturing step further comprises: accessing UV coordinatesof displayed pixels of said 2D snapshot; and generating at least onesurface appearance attribute layer for said 2D snapshot, wherein valuesin said surface appearance attribute layer correspond to values of saidUV coordinates in a texture map of said 3D scene.
 19. The computerreadable storage medium of claim 18, wherein said capturing step furthercomprises naming said surface appearance attribute layer the same nameas said texture map.
 20. The computer readable storage medium of claim19, wherein said modifying step further comprises: identifying pixelvalue changes made in said at least one surface appearance attributelayer; extracting UV coordinates from said coordinate layercorresponding to said identified pixel value changes; identifying saidat least one texture map by matching a name of at least one surfaceappearance attribute layer with the name of said at least one texturemap; and inserting said pixel value changes into said at least onetexture map at said corresponding UV coordinates.